OpenAI analysis flags flaws in SWE-Bench Pro benchmark methodology and known limitations, exposing accuracy gaps in AI coding agent evaluation benchmarks reliability issues.

Claude Opus 4.8 scores 69.2% on SWE-Bench Pro for agentic coding leadership while adding honesty cues and remaining available at prior pricing via EasyRouterIO.